Skip to content

Fix AT-SPI crash and screenshot hangs, add screen capture after sleep - #725

Open
mdshakib007 wants to merge 3 commits into
devfrom
fix-module-and-take-ss-wait
Open

Fix AT-SPI crash and screenshot hangs, add screen capture after sleep#725
mdshakib007 wants to merge 3 commits into
devfrom
fix-module-and-take-ss-wait

Conversation

@mdshakib007

@mdshakib007 mdshakib007 commented Aug 12, 2026

Copy link
Copy Markdown
Collaborator

Three related reliability fixes to the Linux node and the screenshot path, all found while
investigating a node that was producing tracebacks instead of screenshots and periodically freezing
mid-run.


1. Node crash when AT-SPI is unavailable

Every screenshot on a Linux node without the AT-SPI stack produced this, and no screenshot:

TakeScreenShot:CommonUtil.py    Following exception occurred: Traceback (most recent call last):
  File ".../Desktop/Linux/BuiltInFunctions.py", line 15, in <module>
    import pyatspi
ModuleNotFoundError: No module named 'pyatspi'
...
SystemExit: 1

Root cause. pyatspi is a thin Python layer over the GObject-Introspection Atspi typelib, so
the pip side is only half of it — the system typelib (at-spi2-core / gir1.2-atspi-2.0) and a
running accessibility bus are required too. Those are absent on a plain VM whether or not it has a
display. Three defects turned that expected absence into a hard failure:

  1. sys.exit(1) at module import scope. sys.exit raises SystemExit, which derives from
    BaseException, not Exception — so the except Exception: guarding the import in
    CommonUtil.Thread_ScreenShot could not catch it. It escaped to the bare except: in
    TakeScreenShot.
  2. capture_screenshot never needed AT-SPI. It is built on xdotool / xwd / XComposite. A missing
    accessibility stack was killing a screenshot path that does not use it.
  3. It repeated on every screenshot. Python does not cache a module that failed to import, so
    each capture re-ran the import and re-launched the uv add dependency-install subprocess.

Changes

  • The optional imports no longer abort the process. Failures are recorded in ATSPI_IMPORT_ERROR /
    XLIB_IMPORT_ERROR and exposed via is_atspi_available() / atspi_unavailable_message(), which
    names the required system packages.
  • pyatspi falls back to a stand-in whose attribute access raises a descriptive RuntimeError, so
    AT-SPI actions fail as ordinary zeuz_failed steps instead of taking the node down.
  • Action / EditableText / Text fall back to Any — the Accessible protocol evaluates them as
    annotations at class-creation time, so the names must exist.
  • Widened except ImportError to except Exception: pyatspi raises
    ValueError: Namespace Atspi not available when the typelib is missing, which the old code let crash.
  • Screenshots degrade instead of failing: the XComposite path declines cleanly without python-xlib and
    falls through to xwd, and _get_frame_geometry_for_window returns None without AT-SPI so the full
    window pixmap is kept rather than the capture being lost.
  • CommonUtil._get_linux_capture_screenshot() catches BaseException and caches the result, so the
    probe runs once per process rather than per screenshot.
  • server/linux.pyif BuiltInFunctions is None: was unreachable dead code (the import aborted the
    process first). Now uses is_atspi_available(), so /linux/inspect returns the real reason.

2. A screenshot could hang the node indefinitely

Runs would park on the ********** Capturing Screenshot ... ********** line — sometimes for hours or
days — with no timeout able to recover them.

Root cause. Three facts combine into a permanent block:

  1. TakeScreenShot is called outside _run_action_with_timeout, so action_timeout does not
    cover it and _force_kill_hung_browser_sessions never fires.
  2. Thread_ScreenShot is async def, but Driver.get_screenshot_as_file() is a synchronous
    Selenium HTTP call — it blocks the whole event loop.
  3. Selenium has no read timeout: RemoteConnection._timeout defaults to
    socket._GLOBAL_DEFAULT_TIMEOUT, get_timeout() returns None, socket.getdefaulttimeout() is
    None, and nothing in the codebase calls set_timeout. urllib3 waits forever.

A wedged browser therefore blocked the node permanently. The log line is emitted before the capture,
which is why the run always stopped on exactly that line.

Change. _capture_with_timeout() bounds every capture (SCREENSHOT_CAPTURE_TIMEOUT_SECONDS = 60):

  • Sync captures (Selenium / Appium / desktop) run on a daemon thread and are abandoned on timeout —
    the same trade-off _ActionTimeoutWorker already makes for a hung action.
  • Awaitable captures (Playwright) use asyncio.wait_for.
  • Exceptions are re-raised on the calling thread, so the existing except WebDriverException /
    except Exception handlers in Thread_ScreenShot behave exactly as before.

The thread must be a daemon and must not use the default executor: loop.shutdown_default_executor()
joins default-executor threads, so an abandoned capture there would block node shutdown instead.

3. Screen capture after sleep

A sleep is normally used to let a page settle, but the common sleep action declared
"screenshot": "none" — so the moment you most wanted to see was the one moment never captured. The
post-action capture at sequential_actions.py:2789 was already firing; it just had nothing to do.

A common action cannot hardcode "web", because it is shared by mobile, desktop and API runs — a fixed
"web" would make every API-run sleep log a missing-driver warning.

Change. New "auto" screenshot type, declared for sleep and resolved at runtime in
set_screenshot_vars() against the drivers the test actually has open: "web" when a Selenium or
Playwright driver is live, "none" otherwise. Mobile is deliberately not resolved — an Appium capture
on every sleep is expensive and wasn't needed here; it's a two-line branch in
_resolve_auto_screen_capture if wanted later.


Behaviour changes

  • A Linux node without AT-SPI no longer exits; desktop/Linux actions fail as zeuz_failed steps.
    Where AT-SPI is installed the code path is unchanged — every new line lives in an except branch
    that previously ended in sys.exit(1).
  • /linux/inspect returns a JSON error rather than killing the server process.
  • Web runs take one extra screenshot per sleep — slower runs and larger report ZIPs, by design.
    take_screenshot=false still disables it (sequential_actions.py:2734 forces "none" first).
  • A hung capture now costs 60s and a level-2 warning instead of an unbounded freeze. The abandoned
    thread stays parked on the dead connection until the browser is killed.

Testing

Verified on a real node run:

  • The SystemExit traceback is gone; the node ran through the missing AT-SPI.
  • screen_capture = "auto"Capturing Screenshot for Action: Sleep Method: web.
  • The dependency probe and warning appear exactly once per process, not per screenshot.

Automated:

  • tests/test_screenshot_capture_timeout.py (5 tests) — wedged capture times out; the event loop stays
    responsive while a capture is stuck; success path; exceptions still propagate; Playwright awaitables
    bounded too.
  • tests/test_auto_screen_capture.py (6 tests) — Selenium, Playwright, no-driver, torn-down browser
    (stale None keys), and none/web/mobile/desktop passing through untouched.
  • Suite: 111 passed, 1 pre-existing unrelated failure (test_nodejs_appium_installer).
  • Ruff: one finding fewer than baseline (threading is now actually used). No new mypy findings.

Follow-ups (not in this PR)

  • Installer/setup_linux_inspector.sh installs PyGObject's build deps, the X tools and the accessibility
    settings, but its apt branch never installs the AT-SPI runtime itself (at-spi2-core,
    gir1.2-atspi-2.0); the dnf/Alma branch does install at-spi2-core-devel. That asymmetry is the likely
    reason AT-SPI is unavailable on Ubuntu nodes.
  • install_missing_modules runs uv add at import time, which resolves the full dependency set and can
    attempt source builds during a test run. Worth removing from this path.
  • Nodes missing xdotool fail every desktop capture; that is an environment fix (run the installer
    script), not a code one.

@mdshakib007
mdshakib007 requested a review from mahbd August 12, 2026 11:36
@mdshakib007 mdshakib007 self-assigned this Aug 12, 2026
mahbd
mahbd previously approved these changes Aug 12, 2026
@mdshakib007 mdshakib007 changed the title Fix node crash when AT-SPI is unavailable on headless Linux nodes & Take screenshot for Sleep action Fix node crash when AT-SPI is unavailable on Linux nodes & Take screenshot for Sleep action Aug 12, 2026
@mdshakib007 mdshakib007 changed the title Fix node crash when AT-SPI is unavailable on Linux nodes & Take screenshot for Sleep action Fix AT-SPI crash and screenshot hangs, add screen capture after sleep Aug 12, 2026
@mdshakib007
mdshakib007 requested review from mahbd and sazid August 12, 2026 13:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants